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ABSTRACT 

In lesser developed countries, interest has been 
growing in the role evaluation plays in the improvement of 
media-based education programs. Such programs include the use of 
television or radio to communicate a useful message to students, the 
general populace, or a particular group (such as mothers or small 
farmers). The message to be communicate nay concern general academic 
skills (such as reading or math) or sp* ific information (such as the 
advantages of breast feeding or of a particular farming technique). 
In lesser developed countries, such programs are usually initiated by 
political leaders or other high-level decision-makers as a way of 
attacking social problems. In October 1980 a workshop was held on 
methods of evaluating and monitoring media-based education programs. 
This document, prepared as the basic discussion paper for the 
workshop, reviews the rale of evaluation and -^search in various 
stages of the policy-making and implementation process and stresses 
the need to relate evaluation to system planning as a whole, rather 
than confining it to summative "^search or to the formative 
development of materials. This paper has been revised to incorporate 
the major themes of the workshop discussions, including the 
participants' criticisms and recommendations. (Autho 'JM) 
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PREFACE 



In recognition of the growing interest and cc.cern over the proper 
roles for evaluation 1n the .Improvement and expansion of educational media 
1n both formal and non-formal settings, the International Institute for ^ 
Educational Planning (HEP), Paris, and the Learning Systems Institute 
co-sponsored a workshop entitled "Methods of Evaluation and Monitoring 
for Educational Media Systems" on the campus of Florida State University 
from October 27 through 30, 1980. The Workshop brought together twenty- 
seven evaluation specialists and communication planners (see Appendix A) 
for the purpose of reviewing current approaches to the evaluation of 
educational media systems, with the goal of eventually establishing a list 
of priority topics for intensive future study. 

To stimulate discussion and debate among the participants, the 
Workshop's agenda was designed to be as flexible as possible (a copy of 
that agpr.da 1s reproduced in Appendix B). Although papers were not pre- 
sented by the participants, .ix out of the Workshop's eight sessions 
paralleled sections of a discussion paper which had been prepared and dis- 
tributed in advance. Fol lowing the Workshop, the discussion paper was re- 
vised to incorporate the major themes of the discussions as well as the 
participants' criticisms and recommendations. It is presented here as the 
final report of the October meeting. 
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1.0 INTRODUCTION 

Educational media systems make special demands on social researchers. 
These demands also render the perspective of much educational evaluation 
theory inadequate, or at least Insufficient, to the task of illuminating 
how research and evaluation can contribute most effectively to the planning 
and implementation of communication systems. "Isolating priority areas 
for more detailed study" so as to strengthen the relationships between 
planning, implementation ar;d evaluation was the challenge put before 
the authors of this paper and me twenty-five other participants who 
attended the UEP Horksnop convened at Florida State University's 
Learning Systems Institute. 

The bulk of this paper will treat what we perceive to be five phases 
of media system planning, along with the research and evaluation questions 
which accompany them. We will identify those stages shortly, first, 
howeveK we wK; to take up several preliminary issues including: the 
characteristics of media systems which create special research and evalua- 
tion needs in the first place, and a brief justification for the approach 
we will take in dealing with them. In so doing, we recognize that the 
communication system planning approach elaborated in this paper is but one 
of the avenues currently available for both analyzing and projecting 
communication needs and services within a society. Its focus is upon the 
identification of specific objectives and the subsequent application of 
communication resources to meet those objectives. In this sense, our 
approach is highly system specific and project oriented. It differs from 
the alternative, suprasectoral communication planning model which begins 
with an extensive inventory of a nation's communication resources and 
needs and proceeds through the elaboration of a national communication 
plan and, ultimately, to the specification of sectoral objectives ?nd 



initiations (in education, agriculture, health, etc.) consistent with 
the overall plan. 

1.1 Special characteristics of educational media 

The vast majority of all educational media systems may be characterized 
by three factors: geographical distance, extended lead time and institutional 
complexity. 

1) Geographical Distance . The participants in an educational media 
system are typically far removed from the source of the messages they are 
expected to use. Thus, the natural feedback provided by face-to-face 
interactions in traditional teaching and learning situations is simply not 
present except, possibly, for a small fraction of the audience. In addition 
to the logistical problems which this condition creates for program imple- 
mentation, it creates two critical issues for evaluation. One is the need 
to develop adequate mechanisms for getting useful feedback circulating 
through the system. The second is instilling among system planners and 
administrators a sense of responsibility for what happens on the receiving 
end of a communication network. 

Geographical distance tends to diminish the importance of pedagogical 
criteria in the assessment of program impact. Since audience effects 
happen at a distance, there is some tendency to equate effectiveness with 
artistic quality of the system's media products - television programs, 
radio soots, brochures, etc. This tendency also weakens the planners* 
concern with implementation and utilization processes, causing them to 
view such processes as essentially outside their control. By ignoring 
forces which affect the unfolding and eventual success of programs in the 
field, media planners are open to the accusation that their perspective is 
overly centralized, non-participatory in nature, and unreceptive to local 
initiative and control. 
• -2- ( ^ 
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7) Extended lead time . The calendar of planning, system design 
and program preparation (and of the investments which characterize the 
start-up phase of a media project) is radically out of phase with the pro- 
posed calendar of program utilization. Planning and program development 
are heavily concentrated at the beginning of a project. Utilization 
activities, on the other hand, are concentrated in later phases. While all 
educational programs live with this problem to some extent, media projects 
exhibit it in the extreme. There are two telling reasons for this. First, 
many media projects ent'il substantial capital investments which must pre- 
cede the initiation of a program. Once the investment in a particular 
technology is made, other options are essentially foreclosed. The precedent 
of early technical decisions is great because of the difficulty of reversing 
or redirecting large capital investments. This may place a special burden 
on researchers to inform policy-makers of the likely consequences of such 
decisions before direct evidence about their consequences is known. Secondly, 
one of the strengths of media-based education projects is their ability to 
amortize software investments over a number of years. The expectation that 
a particular instructional unit or package can be used again and again 
justifies heavy development costs. However, here, too, an inevitable time 
gap develops between program preparation and utilization. While this may 
also be a problem in programs utilizing other instructional aids (textbooks, 
for example), media-based projects typically depend more heavily on prepared 
materials and are thus more vulnerable to the aging process. 

The time gap between program preparation and utilization, like that 
between investment decision and utilization, has the consequence of sub- 
stantially reducing flexibility within a media system. Implications for 
evaluation of that reduced flexibility are two-fold. On the one hand, ic 
demands that evaluators be realistic about the amount of change that an 



already operating system will make in the short run. On the other hand, 
it suggests a need to concentrate evaluative efforts on those areas of a 
project which are genuinely open to change. 

3) Institutional complexity . Educational media projects, if they • 
are to be successful, inevitably rely upon the coordination and active 
cooperation of multiple bureaucratic layers and constituencies. The programs 
may also involve multiple communication channels, a process that frequently 
necessitates collaboration among public agencies or between public and 
private agencies that have never before worked together. They often bring 
together individuals of sharply different backgrounds (artists, civil 
servants, social scientists, engineers), and cemand that such people work 
together productively. Such institutional \r % i professional complexity 
creates special opportunities as well as specia. problems for researchers. 
In the maelstrom of diverse loyalties, backgrounds and responsibilities, 
the possibility of tension among people and departments is high. Break- 
down in coordination, and readiness to assign blame for those breakdowns 
to others, must be expected. Here evaluators walk a tightrope lest they 
become overly identified with one group or another, and thereby risk 
ostracism by the others. On a positive note, the complexity of educational 
media projects may be substantially reduced by an internal evaluation system 
which provides mechanisms for the exchange of information among people 
whose activites must be coordinated. 

1 . 2 A Perspective on Evaluation 

Social scientific research can contribute in many ways to the improve- 
ment of educational media systems, but only a small fraction of them fit 
under what is usually associated with the term "evaluation/ Historically, 
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most evaluation efforts associated with educational media have dealt with 

long range issues oV system effectiveness an^-eos.t. In other words, did 

f 

this or that project or system accomplish ^what it set out to do, did it 
fulfill its objectives? Such a perspective unquestionably adds to our under- 
standing of particular projects and, cumulatively, if associated with process 
information, to the wisdom of investing in media programs generally. How- 
ever, it is of little use to the projects themselves. As Hornik (1980) has 
written: 

In order to make a reasonable judgment aboilt whether a given 
project is a success, the evaluator must wait uncil it has reached 
a relatively smooth operational stage. However, by that time 
substantial financial and political commitments have been made 
and are essentially irreversible. After generating the momentum 
to begin a program of serious educational broadcasting, convincing 
the various elements of the community to forge ahead, building 
the studios, and installing the transmission equipment, no one 
will be much interested in finding out that the project does not 
work. A negative report will be buried faster than it can be 
duplicated, unless a change in the political environment opens 
the system to new information. 

In considering the value of research and evaluation to ongoing systems, 
we must first, recognize some general principles* There are a very large 
number of decisions to be made. The character of those decision? changes 
as the project matures from its earliest planning stage throt^h full imple- 
mentation. Some of these decisions will be made as commands; others will 
be negotiated among various constituencies. The researchers' leverage 
will vary according to the poli tical/tecnnical context of the decision and 
to the feasibility of gathering specific data pertaining to it. Together, 
these conditions (and they are a small subset of a larger group of closely- 
related conditions which can be found in Cronbach et al.(1980) offer 
guidelines for the conduct jf evaluation. When decision frameworks change, 
then so, too, must evaluation perspectives and concerns. Many of these 
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concerns will not he evaluative, in the sense of rendering judgments about 
project consequences.* Rather, they will be aimed at gaining a better 
understanding of the environments in which projects operate. 

In the sections that follow, we argje that the? utility of evaluation 
to educational media planning and performance ^ests on the willingness of 
a variety of actors, decision-makers as well as researche , to assess the 
context in which decisions are taken, and to plan evaluation activities 
accordingly. In describing important research issues for educational media 
evaluation, we use the shifting roles of evaluation as an organizing device. 
Accordingly, we concentrate on the roles for research and evaluation in 
each of five phases of project development - policy definition, system 
design, system build-up, system maintenance, and system review. For each 
phase we identify key activities and decision-makers, assess the variety 
of information needs that are likely to emerge, and specify areas ir, which 
we'feel additional investigation is called for. The paper closes with a 
brief section that sunmerizes additional concerns and qualification raised 
during the workshop. 



^he term "evaluation 0 is frequently used to denote any research 
having to do with social interventions, and not just those special kinds of 
research which attempt to place a value on the results of those interventions 
or their constituent parts. However, we believe that only a small percentage 
of social\esearch activities are genuinely evaluative - in that they attempt 
to render judgments 0f worth. While the boundary between evaluative and non- 
evaluative research may be fuzzy, it is, we believe, a distinction that 
ought not to be ignored. 



2.0 POLICY DEFINITIO N 

2 . 1 Who are the Actors? 

The planning of new or exoanded communication services is often begun 
by individuals with little working knowledge of the media or even of the 
specific development services they wish tc improve. Frequently, the j 
process is initiated by political leaders or other high-level decision- 
makers (planning commission members and the like) whose responsibilities 
transcend individual government ministries. As highly sensitize political 
actors and as general ists lacking detailed knowledge of the social issues or 
problems they wish to address, these individuals may be attracted pre- 
maturely to media schemes which promise quick solutions. The appeal of 
"technology-oriented" (as opposed to "problem-oriented") transfer activities 
is consistent with the desire of such decision-make. * to launch highly 
visible and politically appealing programs. 

2.2 What decisions are involved? 

Initial planning activities - assuming they incorporate all'decisions 
up to ana including the promulation of an official policy and the allocation 
of specific resources - have two important consequences. First, they 
establish political boundaries for subsequent planning stages and activities. 
While other actors and forces may come into play later, either enhancing 
or reducing the effectiveness of particular systems as they approach full 
scale operation, the essential political character of the enterprise is set 
during the policy definition stage. Secondly, the decisions made here in- 
evitably delimit the range of system options including goals, implementing 
institutions, target audien:es s operating mechanisms and timetables. If 
such concerns are inadequately explored, irreparable damage may be inflicted 



on a communication system before it has even left the drawing board. 

In general, given the actors involved, one hopes to maximize the poli- 
tical commitments achieved at this stage while, at the same time, minimizing 
the constraints placed on system design. Obviously some aspects of project 
design will not be left open, since they wilV>rtK(e become a part of the poli- 
tical commitment itself.- Mass media, for example, may be incorporated at the 
outset because they promise the 'n'gh visibility political sponsors seek- 
Likewise, certain institutions may be granted a central role in project ad- 
ministration because they might otherwise resist it. The direct replacement 
of existing field staff with a media channel may be ruled out so as to 
neutralize the potential opposition of that group. The essential question 
for us at this stage is what kinds of information will have leverage on 
policy decisions, and how best can that information be gathered and pre- 
sented to decison-makers? 

2.3 -" What information is required? 

Ideally, the desire to develop or enlarge communication capacity within 
one ormore development sectors should flow from the identification of con- 
crete social needs and aspirations: the failure o f a school system to offer 
adequate natural science instruction at the secondary level, the inability 
of an agricultural extension service to reach small farmers on a regular 
basis, etc. The list of needs may seem virtually inexhaustible, and the 
first requirement of policy planning may be the establishment of priorities . 
among competing social needs. However, the choice among social needs - 
which is really a choice to satisfy one constituency rather than another - 
is primarily political, and in any case not significantly a communication 
policy decision. It is at the next level of policy concern - how to satisfy 




a selected social need - which is our first concern. 

Information requirements at the policy definition stage correspond to 
the types of policy decisions previously described. The first requirement 
involves assessment of the specific development problem to be addressed. 
That oegins with a descriptive analysis: How widespread is the problem? 
What are its manifestations? It then turns to an understanding of causes 
and an identification of why previous solution attempts have failed. 

Is there agreement among interested parties regarding the exact nature 
of the development problem to be addressed? For example, to the extent 
that an agricultural extension service is failing to serve farmers in a 
* satisfactory way, where does the fault lie? Is it with the users themselves 
who may lack the knowledge and/or motivation to use available services? 
Could it be that the field staff lacks the experience or training necessary 
to work effectively with the clients in question? Could it be that the 
program's central administration is not backing the program with sufficient 
material or human resources? Or could it be that the essential program 
assumption - that there is information worth disseminating - is wrong? 
One or more of these shortcomings could account for failure and investigation 
may be required to determine where the true problem(s) reside. By rushing 
into a solution incorporating communication media without systematically 
examining the kinds of questions posed above, planners risk developing 
elegant (and expensive) solutions to insignificant or inadequately understood 
problems. 

Regretably, a premature focus on the potential for resolving a particular 
problem using the media has often prevented decision-makers from adequately 
examining the scope and or. gins of the problem itself. This has led to 
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countless feasibility studies whose purpose more often than not has been to 
examine the viability of particular media-based communication strategies 
rather than the factors underlying a specific social problem and alternative 
ways of dealing with it. Until planners are encouraged to probe the under- 
lying social, economic, cultural a.id/or psychological foundations of the 
attitudes and behavior they wish :j alter within some target population, it 
is likely that their judgments will be predicated largely upon their personal 
intuitions, the advice of knowledgeable but unfamiliar foreign experts, and 
simple political expediency. 

If policy planners can satisfy themselves that the problems they have 
selected to work on are genuine, and that such problems are amenable to 
certain kinds of treatment, they may work to develop broad project goals, 
responsibilities and timetables. In the case of educational media programs, 
the identity of a target audience and the preferred means for reaching that 
audience on a regular basis may be specified also. Information about the 
potential of alternative project configurations, about the reasonableness of 
goals, and about the time delays typical of such projects, is apt to be 
partiuclarly important. 

At the policy definition phase, there is a special need for information 
that will support institutional planning. Communciation projects character- 
istically involve more than one government ministry or agency. How close 
do the established mandates and programs of those institutions correspond to 
the emerging goals of the communication initiative? In other words, do the 
leaders of these institutions share the perspective and concerns of the policy 
planners? What are the forces which work, for and against long term institutiona 
collaboration? Will the promulgation of new programs or projects necessitate 
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the re-allocation of existing budgets, or will new funds be forthcoming to 
support such activities? Given the hierarchial relationships likely to be 
Evolved, how can the policy planner be certain that the expression of 
institutional interest is genuine and not simply the sign of token obedience 
which will not necessarily be translated into effective action down the 
line? Where should management responsiblity for the envisioned program(s) 
be located? What role can coordinating bodies hope to realize? What 
criteria will be used for allocating resources, both for the detailed 
planning effort and for the actual administration of the project? 

While it would be ideal if a pre-designed project could be assured 
of a perfect institutional niche, the reality is that most projects adapt 
to the strengths and weaknesses of existing agencies. The question, there- 
fore, becomes what aspects of the alternative implementing agencies are most 
likely to help or hinder system design and implementation? Are competent 
staff available to undertake innovative projects? The answers to such 
questions are rarely, if ever, self-evident, and frequently dangerous 
shortcuts have been taken in order to comply with funding deadlines or the 
desire to announce some new program at a politically propitious moment. 

2.4 Data sources 

Adequate assessment of public policies and of the related administrative 
issues which are likely to emerge as part of any planning initiative depends 
on reliable information, information which can come from a variety of sources 
What are the sources relied upon by high level communication planners? How 
dependable and useful have such sources been historically. Among 204 decision 
makers in "policy influencing positions" within the U.S. government, 
Nathan Caplan (1979) found a pronounced eclectism in the search for policy 
guidance. Policy decisions, he found, were influenced by "information 
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acquired independently from diverse sources external to government - news-* 
papers, books, professional journals, magazines, television and radio, "(p. 465). 
When dealing with related administrative issues, however, the decision- 
makers were more parochial, relying almost exclusively on internal governmental 
reports and memoranda. Within each domain Caplan found a "great sensitivity" 
to interpersonal (and informal) sources of information. The dependence on 
such sources far ou;weighed the attention to empirical data. 



2 . 5 Questions for the Research Agend a 
2.5.1 Institutional Questions 

So far we have assumed that (1) there is a discrete policy definition 
phase in communication planning, and (2) there are political actors involved 
in this phase whose decisions are open to information and thus are not 
entirely defined by the political process. The confirmation of these assump- 
tions - or more accurately - the specifications of the conditions which make 
them more or less realistic, is the first order of business. 

1. What are the range of decisions that are customarily made in 
the period before a project is officially in existence? Which 

of those decisions are typically fixed, and which open to modifi- 
cations at a later project stage? Which decisions are politically 
mandated, and which open to research information? Who actually 
makes communications- relevant policy? 

2. What types of information are both feasible to obtain and 
convincing enough to exert leverage on decision-makers at this 
stage? Do individuals engage in policy work themselves feel the 
need for data to guide their deliberations? What information do 
they now get, and what additional information would they think 
useful? What sources of information would they consider credible? 
What time and budgetary constraints operate during this planning 
phase which will influence the feasibility of alternative data 
collection efforts? 

3. Do the answers to the previous questions vary with different 
sectors, with the character of governmental authority and overall 
development policy, with relationship to foreign and multi-lateral 
aid and technical arsistance agencies, and with the degree of 
collaboration across governmental agencies required? More generally, 
is each project environment essentially unique orcan some cross- 
project statements be made which suggest when information will be useful? 



Research strategies for investigation these questions might include one 



or more of the following: 

a. Administrative histories of communication projects which would 
utilize interviews with planning commission officials and others 
knowledgeable about the creation of specific programs. Such inter- 
views would be used to reconstruct the influences and timing of 
major decisions an attempt to provide answers to each of the 
above questions. An expensive version of this project would re 
quire a large number of interviews in each of a large number of 
projects. A less expensive version would require interviews with 
only one or two informants per project, beginning with relatively 
few projects. Additional projects would be examined only if ther* 
Is some pattern to emerging results (not entirely unique to each 
case) and until there is substantial redundancy in findings (so 
that additional investigation would be unlikely to add very much 

to our understanding.) 

T\e major risk with this strategy is tautological (e.g., the tendency 
to conclude that "what is, is inevitable"). If there is no instance of 
planners having easy access to rich and relevant information prior to making 
their decisions, it may be difficult to guess what difference that information 
would have made if it had been available. That is true whether or not 
planners testify that they would have used such information. In addition, 
it may be difficult for informants to recognize and thus articulate the in- 
fluence of background knowledge - shared assumptions amo j planners never 
made explicit - on decisions. 

b. A complementary research approach, designed to describe this 
unobserved circumstance, is more direct. Rather than writing 
administrative histories, it would entail providing l numbe^ of 
planning commissions with just such a research competence, and 
seeing what the result is. While such efforts would have to be 
relatively few, and thus make separation of the unique aspects 

of a given project from more general izable results tentative, they 
might prove instructive. 

c. Finally, the old stand by - the literature review - may prove 
helpful in looking at these questions across different sectors. 
While only some of these lessons might be directly relevant to 
communication programs, the effort could provide useful guidelines 
for future studies. 



ERLC 



-13- 

19 



2.5.2 Methods Question s 

We have described four types of data that may be helpful to decision- 
making in the policy definition phase: (1) description of the exten- 
siveness of a particular social problem, (2) the etiology of the problem 
and its likely susceptibility to communication-based solutions, (3) likely 
consequences of alternative project configurations that can be envisioned 
at this stage, and (4} probable consequences of alternative institutional 
locations for projects. How to obtain the first of these, evidence of 
extent, is often not problematic and may not present a special problem for 
communication-based projects. Methods chosen will have to respect the special 
time constraints of this phase, however, but that is an issue that will be 
treated subsequently. The third area, dealing with the consequences of 
alternative project configuration really incorporates two sub-issues: what 
sort of information will be influential with policy-makers (already t> 3ated 
as Research Question #2 above) and what ways are there to predict the con- 
sequences of a particular project design - which will be the major research 
question of the next section. The essential research methods questions of 
this phase which remain are: 

1. How does one understand the economic, social, cultural and 
psychological forces which maintain a particular behavior? In 
particular, how does one know whether that behavior is alterable 
by communication efforts, either in isolation, or in combination 
with new resource inputs. 

2. What are reliable approaches for defining feasible institutional 
roles for various existing and newly-created agencies in the plan- 
ning , development and long term optr^Hrn of a particular project? 
How does one define the circumstances when single institutions 

are appropriate, and when multi-agency collaboration is required? 

As before, answering these questions will require a number of research 
approaches. However, all of them are constrained by two assumptions. First, 




■14- 



20 



time is likely to be short in this planning phase. In some cases no original 
field data collection will be possible. In virtually all cases, lengthy 
field studies (multi-year ethnographic studies, for example) will be in- 
feasible. Secondly, expectations for a wel Vspecified a prion behavioral 
model must be realistic. Social researchers never completely understand a 
behavior. A more reasonable goal is to specify whether a change in a be- 
havior has some promise of being associated with a particular communication 
intervention. 

In a sense it is misleading to focus on methods for defining a be- 
havioral mode. The methods one can apply are reasonably well known. The 
question of interest here how one organizes the application of jsome or 
all of the existing methods (literature review, expert advice, surveys and 
other ethnograpnic studies) in support of the planning process. One must 
develop procedures to make explicit those assumptions about how a behavior 
fits into its environment which are inevitably present in a proposal to use 
cotnmuni cation media for social change^ The process itself may make the 
logical weaknesses of a program apparent. Secondly, the search for evidence 
concerning each of those assumptions must be commissioned so that relatively 
lower cost data collectijn efforts (literature review and expertise) are 
exploited before turning to relatively higher cost (in time and money) field 
data collection. This 1s particularly import:- 4 - because the rather sophisti- 
cated research design skills entailed in directly testing assumptions may 
be beyond the resources of many programs. 

a. There are few examples where the statement and test of a 
behavioral model has preceded the decision to go ahead with a 
communication plan. Thus historical case studies are unlikely 
to yield much information about how to do this. A better result 
is likely to come from studying the process directly by working 
with a number of planning groups. 

ERIC 21 



Institutional analysis relies on three activites - a clea^ specifi- 
cation of just what tasks must be undertaken in the planning and implemen- 
tation of communication systems, a bringing to bear o, prior international 
experience with alternative institutional arrangements for realizing a 
project, and a clear-eyed evaluation of institutional tensions and person- 
alities in various local situations. These last two activities may suggest 
research studies worth pursuing, but they do not seem to fall within the 
mandate of this project. It is only the first, the specification of tasks 
to be undertaken, that would seem to make demands ir, the research and evalu 
ation function of projects. 

b. The problem here is to try to get planners to recognize just 
what activities they are expecting will be undertaken and, sub- 
sequently, to identify what institutional resources and what 
levels of collaboration are implied by those activities. Again, 
the paucity of cases where this has occurred suggests the value 
of a direct intervention to both implenient and study the effect 
of alternative task analysis schemes. 
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3.0 SYSTEM PLANNING / 

3.1 Who are the acto rs? 

Once the general outline of a communication poTicy has been defined, 
the responsibility for implementing that policy customarily passes from 
the high-level planning group to persons who will actually oversee the 
development of a workable system and who will ultimately be held account- 
able for its success or failure in the field. Such individuals frequently 
hold high civil service positions 1n the particular government agencies 
t ose programs are to be expanded and/or improved. They are often found 
within national broadcasting authorities as well. 

The training, experience and concerns of system planners are apt to 
be parochial and based upon experiences within single development sectors 
(education, health, agriculture or broadcasting). By the same token, such 
individuals are also more likely to understand the practical problems of 
mounting any new program in the field, and the resources required to do 
so, than are the general ist decision-makers described in the previous 
section. Agency officials may be resistent to any policy changes that 
have been decreed from on high, particularly if such changes are not the 
result of at least some consultation between the two levels. In -,hort, 
for ownership to pass efficiently and enthusiastically from one planning 
level to tha other, the leaders of the "user" agency(ies) must believe 
(or be persuaded) that it is in their interest to cooperate and that 
their desires and constraints have been adequately considered in the 
formation of new policy. 
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The development and implementation of a truly innovative communication 

program is likely to require a considerable rethinking and re-evaluation 

of past policies and procedures, however, and it is in this area that 

the experience and conventional wisdom of agency leaders may be a hindrance 

as well as a help. Fervent disagreement, or at least uncertainty, 

be provoked as policy is translated into workable program practice, 

but unless such disagreement or uncertainty is allowed to surface and 

be resolved, there is danger that the new policy will be crushed by the 

weight of bureaucratic precedents and assumptions. For this reason, it 

is essential that system planners be granted the time and resources to 

develop a comprehensive system plan, a plan based on fresh empirical 

data from the field as well as the accumulated experience and wisdom of 

officials at various levels. At the same time, gatherersof fresh empirical 

data must be aware of important limitations on their influence. We quote 

from Hornik (1980) 

Evaluation operates in^a political environment. Decisions 
about social programs generally, and about the application 
of educational broadcasting specifically, are reached through 
a complex process. Whatever the actual chain of command, 
whether the society be democratic or authoritarian, the decision 
about whether and how to implement educational broadcasting 
will be negotiated among competing constituencies. These might 
include the planning staff of the person who signs on the 
bottom line, the entrenched educational bureaucracy and its 
network of power relationships, teachers, students, parents, 
the business community and other employers, foreign aid agencies, 
and other governmental entities. 

Each of these concerned parties will attempt to influence the 
shape of the decision. Each has its own values to maximize 
and expectations of how much influence-it will be able to 
wield. Decisions taken in such a political maelstrom are not 
likely to hinge on cost-benefit ratios. 
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Further, there will be no single decision. Even if there *s a 
commitment to go ahead with educational broadcasting, the shape 
of the program will be hammered out in negotiation among the inter- 
ested parties. In that process, there will be room for Information 
an evaluation can supply; there will also be major decision areas 
that will be so constrained by political concerns that the evaluator's 
efforts would be irrelevant. 

The evaluator can help the decision-making process in this context 
only if he u^i stands the political forces at work. On the one 
hand, he can determine which parts of the decision are open to 
information - thore in which political interests are not at issue - 
and gather the appropriate data. For example, research evidence 
about tne amount of time a child will pay attention to a program 
will almost surely weigh heavily in decisions about the length 
of in-school instructional radio programs. 

Beyond gathering information to affect technical decisions, the 
evaluator can work in the political arena as well. 3y providing 
credible information, the evaluator can help interested parties 
negotiate with a clearer picture of the likely consequences, and 
thereby surely improve the political process. It should be empha- 
sized that the evaluator will be helpful only insofar as he gathers 
information at points of leverage in the system, that is, about 
those issues that have not been decided upon in advanc3 of negotia- 
tion. 



3.2 What decisions are involved? 

Identification of the audience or client group . Of prime importance 
to any communication plan is the preparation of an adequate audience profile 
Who are to be the participants and beneficiaries of the new or expanded ser- 
vice? How do they assess their own problems and needs? Are they readily 
available - in schools, clinics or cooperatives - or will substantial effort 
have to be made to reach them, either in groups or individually? How famili 
are they with the information likely to be contained in the program? Can 
interest be assumed or will it have to be stimulated and channeled by the 
program? What is known about the way a particular audience is likely to 
receive and ideally respond to a communication system? 
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In practical terms, how great a disposition ta change is there within 

potential client groups? tfow compatible are existing attitudes and behaviors 

within the audience with the content which is likely to be presented? 

What factors sustain such attitudes and oehaviors at the local level? 

Many of these questions would have been addressed also in the previous 

stage but in lesser detail. 

Selection and-fcfllaneg^of communication channels. Having assessed the 
— ■ 

availability of a potentiate! ient group and the strength ^f its existing 
attitudes and behaviors, the way is clear to consider what combination of 
communication channels will be employed to reach that audience. Even in 
those instances where an a priori decision has been taken to make some 
medium, say rajio or television, the primary means of instruction (the 
"master medium" in Tiffin's and Coombes* words), the selection of and 
investment in supplementary communication channels can be as critical a 
planning decision. Such a decision must be based on a variety of considera- 
tions including: the access of potential users to particular media, their 
prior experience with and- attitudes toward such media, and the ability of 
the sponsoring agency to provide supplementary channels on a reliable and 
sustained basis. 

Messa g e design and presentatio n. A similar set of concerns arises 
in the selection, ordering and actual presentation of the information which 
different channels will be expected to carry. Knowledge of audience abilities, 
habits, expectations and needs coupled with a clear set of communication 
objectives (e.g., will the intent of the program be to persuade as well as 
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inform, to instruct as well as motivate?) constitute the critical bases 
upon which preliminary content decisions can be made. From such decisions 
should also flow guidelines to help content experts and production personnel 
(writers, producers and directors) establish tentative norms for the length, 
frequency, complexity and pacing of their message. 

Message reception and utilization . How members of the audience receive 
and use new information at the local level is likely to be as Important a 
determinant of program success as the appropriateness and appeal of the in- 
formation Itself. Questions here encompass a broader institutional focus 
than those of physical access already discussed. They stem from our as- 
sumption that communication services must complement the provision of mecha- 
nisms for the expression and realization of new knowledge, attitudes and 
practices. Project sponsors must be prepared to deal adequately with success 
or be sure that someone e.se will. Does the agency have the requisite re- 
sources in place to respond to enhanced client demand? How quickly will it 
be able to respond to heightened expectations? The answers to such questions 
suggest the need for a careful analysis of existing response capability as 
well as a projection of the costs and delays likely to be incurred in ex- 
panding such capability. 

3.3 Where can the information be found? 

Answers to the variety of questions posed above are most frequently 
found within the sponsoring agencies themselves. They may be contained In 
officially commissioned studies or in administrative memoranda and intra- 
agenc^y reports. Such sources tend to be self-serving 1n administrative 
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tenifs^and they rarely depart from conventional modes of defining a particular 
organization's constituencies - school teachers and administrators, doctors 
and health workers, agricuUuraJ extension agents, etc. Also, when such 
documents deal with the supposed beneficiaries cf the respective services, 
they tend to deal in strictly quantitative „2rms. Rarely is any attempt 
made to probe qualitative differences among the client group, much lees to 
relate those differences to current needs. 

A second source that is frequently relied upon to guide system planning 
is outside experience and expertise. This can take. various forms, from a 
simple review of projects elsewhere the full incorporation of experts 
from various bilateral and multilateral agencies in the planning process itself 
These strategies permit the planners of new systems to make use of conven- 
tional wisdom on a world scale, and they are likely to prevent repetition of 
the most glaring errors that have beset other projects, t t they are un- 
likely to instruct planners as to what -ill work best in their own environ- 
ment. 

Still another strategy for obtaining vital planning information in- 
volves mounting a prototype or pilot version of a system one would hope to 
see enacted eventually on a large scale (example: Nicaraguan Radio Math). 
Such an approach has many benefits in that it theoretically permits both 
strengths and weaknesses of any plan to be identified before it is imple- 
mented on a broad scale. Such efforts can be closely monitored so as to 
determine the right combination of resources necessary to meet a specific 
objective. In the final analysis, however, prototype or pilot projects 
are usually unrealistic from a planning standpoint. As much effort and 

-22- 



9 

ERJC 



28 



money may be required to mount the pilot as to launch the first stage of 
an operational system, and many planners simply do not have the flexibility 
or budgets to undertake what are essentially experimental programs. 
Such pilot programs, given the political and financial commitments required 
to initiate them, may be as unmodifiable as projects which start up without 
an identified pilot phase. Also, there is real risk that, the special condi- 
tions arranged for a pilot program make them unrealistic tests of how opera- 
tional projects would survive under what are invariably less favorable 
conditions. 

The problems surrounding pilot p-ojects illustrate the kinds of trade- 
offs whicn educational media planners inevitably confront at the system 
design stage. Planners are faced with a multitude of decisions to make, 
which, once made, set 1n motion policies whose character and momentum may 
be very difficult to alter later on. The need for informed decisions is 
acute'. At the same time, planners may be operating within a political man- 
date whose direction and timetable are already set. Under the latter 
circumstance, research of any kind may be viewed as at best a diversion 
and at worst a waste of resources or as footdragging by the program's initial 
advocates. Are there ways in which the common rush through system planning 
to implementation can be tempered to the benefit of all concerned? 

The planners' desire to enact new projects as expeditiously as possible 
may also impede their communication with and understanding of the very indivi 
duals or groups they hope to influence. Much has been written in recent 
years about the non-participatory nature of planning and of social research. 
Hall (1978) and others have adequately documented the shortcomings of the sur 

-23- 

29 




vey instruments researchers have used to diagnose various educational prob- 
lems and-rveeds. What has been missing in planning and in social research, 
according to the critics, is a willingness on the pa*t of investigators to 
share their findings with their subjects or to let the latter play a mean- 
ingful role in the analysis of their c situation and needs. 

But to "expand the framework for planning beyond the safe haven of 
government officialdom and 'experts, 1 bu^h native and foreign, is to intro- 
duce problems and delays which most planners find intolerable," according 
to Mayo (1980). "First, there is the problem of representativeness: Who 
is to speak for the evencual recipients of the envisioned communication 
service? Second, there is the problem of consensus: Even if representa- 
tives of the potential client group can be identified, do they know what 
they want? Third, there is the problem of efficiency: Will not the costs 
and delays be increased too much by trying to broaden the planning frame- 
work?" We can add here the problem of disinterest; the institutional 
rewards for stimulating participatory planning may be few. Satisfactory 
resolution of such issues may require substantial investments of both 
time and money. Yet failure to identify and strengthen the bases of popular 
support for any new program at the system planning stage may seriously re- 
duce that program's chances for eventual acceptance and success. 

3.4 Questions for the Research Agenda 

Insofar as planning issues have been incompletely resolved in the 
policy definition phase, they must be treated in the system design. The 
central question in the system design stage continues to be the definition 
of a behavioral model. Although the actors are now different, questions of 
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institutional responsiveness to information remain criucal. However, 

there is a new aspect of institutional responsiveness which can be raised 

at this stage, and it is the first of our research issues. 

1. Participation, as a means and end of communication planning, 
is a concept which begs for explication and analysis. In short, 
what range of processes and outcomes do people have in mind when 
they call for more participation. Participation by whom, and for 
what? In terms of educational media, specifically, in what ways 
can planners broaden the base of their deliberations? What kinds 
of linkage arrangements can now be proposed to facilitate communi- 
cation between planners and local groups to assure that the 
former are acting responsibly in the latter's behalf and that 
the latter are adequately prepared to accept and ultimately pro- 
fit from new communication initiatives? 

2 A subsidiary research question points more directly at evaluators. 
-We have argued (consistent with Cronbach, et al . 1980) that success- 
ful evaluation is evaluation that adequately mforms each consti- 
tuency with a legitimate interest in a social program so that it 
can negotiate knowledgeably for its interest. Just as one wants 
to be able to specify participatory planning strategies, partici- 
pation by each constituency in the definition of what evaluators 
are to study is also worth striving for. How one does this and 
what impact it has on project design is a vital research question. 

Specific strategies for answering each of these research questions might 

include the following: 

a. An attempt to define just what is meant by participation with 
regard to communication projects, and wha^ the results of such 
participation are expected to be. A review o* the essays which 
argue f or this is a first step. Greater benefit could probably 
be achieved by examining, at least second hand, the organization 
of projects deemed participatory by their supporters or other in- 
formed observers. 

b A second approach to participation would involve a sustained 
examination of reported examples of participatory planning, both 
in more and less developed countries. Are there instances in 
which projects have successfully incorporated representatives of 
the projected client group in the planning process? Alternatively, 
are there instances in which the initiative for planning has 
emanated from local groups and subsequently been amplified through 
the incorporation of specialists from outside the local arena? 
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c. A third strategy, deriving from the first two, would involve 
the insertion of promising participatory planning methods into 
situations in which leaders were receptive to such innovation, 
arid then evaluating the results. 

Participatory planning strategies 'nay provide useful guidance to system 
planners or they may - and this i? the more likely outcome - clarify just 
what the key planning research questions are for the several constituencies, 
while not providing complete answers to those questions. The next step, 
whether or not preceded by participatory planning, will be research aimed 
at specific questions whose answers will guide design decisions. 

3. What is the objective validity, precision and cost associated 
with alternative methods for obtaining information? Each of the 
information gathering procedures available to communication planners 
at the system design stage - survey data, results of pilot studies, 
experienced informants, outside experts and client group participation- 
can produce answers to most of the planning questions listed above. 
However, each answer will vary according to its validity and pre- 
cision, its credibility to planners and other constituencies, and 

the resources (time, budget and expertise) required to produce it. 
As we have already stated, the contribution that information of 
a given kind and quality will make to planning will be constrained 
by the idiosyncracies of each situation. Nonetheless, knowledge of 
the quality and cost of alternative strategies for answering parti- 
cular questions will be very helpful both to planners and to re- 
searchers. 

4, Many projects gather reams of data as a guide to planning. 
For most, however, it is difficult to detect any important effect 
of the information in actual planning decisions. Why is that? 
What research planning process is likely to improve the definition 

of research questions, so that the relation between the data gathered 
and the decisions to be made is well understood? 

Research strategies for answering these questions might include one 

or more of the following: 

d. Case studies of design decisions across a variety of educational 
media systems would help to discern patterns of use of alternative 
information sources. Satisfaction with available information and 

a description of unmet information needs might also come from in- 
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depth interviews. However, it will be difficult to separate dis- 
satisfaction with inadequate methods from dissatisfaction resulting 
from poor procedures for 1 inking decisions to research activities. 

e. Work directly with new project planners and researchers, building 
on whatever was learned from the case studies to test promising pro- 
cedures for linking decisions to research activities. 

f. Provide additional funds and technical assistance to a small 
number of projects (simultaneously with (e) above) to test altera- 
tive methods for gathering information and using it in the planning 
of educational media projects. This should be preceded by extensive 
review of existing experience with alternative research procedures, 
both in the published literature and in the experience of major 
communication projects. Questions of cost in time and budget, re- 
quired evaluation expertise, validity and precision, and of the ability 
to describe variation as well as central tendency would each be 

worth studying. 

As a closing note, we recall that no matter what the amount of data 
available, it is inevitable that major decisions will have to be made with- 
out complete information. Ironically, when such information finally does 
become available, major project decisions have alraady been made and are 
probably irrevocable. Despite this condition, cross-project investiga- 
tions of the kind suggested above could go far to reduce the magnitudes 
of uncertainty surrounding decisions at the system design stage. 

The major result of the system design stage and the social research 
strategies contained therein should be a comprehensive plan consisting of: 
a target population, a list of key attitudes and behaviors of that popu- 
lation which are to be addressed, a communication strategy to influence 
those attitudes and behaviors, and a timetable for identifying, preparing 
and putting to use all the resources necessary to accomplish the original 
policy objectives. This is a tall order and one which has rarely, if ever, 
been met by planners to a level of specificity to prevent major adjustments 
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from occuring once the program itself has been implemented. Experience 
suggests that major revisions will and shoul d occur within all the above 
areas as a nev communication system is subjected to the demands and strains 
of the real world. Given this fact, it is appropriate to judge system de- 
Signs not only by their thoroughness and by the fidelity with which they are 
eventually implemented, but also by the models of change they posit and 
by the guidance they provide administrators and evaluators for testing and 
modifying the critical assumptions and relationships contained in the 
models themselves. 
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, 4.0 SYSTEM BUILD-UP 

4.1 Who are the actors? 

Once planers within the agency charged with system design complete 
their work, they customarily turn over responsibility for implementation 
to a project staff. Ideally, the planners will turn over a detailed system 
model, including a rationale for why and how a successfully functioning pro- 
ject will actually affect people in the field. In fact, it is useful to 
divide the system design phase from the build-up phase at the point that 
such a model is specified, although we recognize that in many cases *he 
detailing of the model will be left in the implemented 1 hands. 

Assuming that the behavioral model and the operational model have been 
adequately spelled out, it is the researcher's job, in collaboration with 
the project administrator, to verify components of the model as the system 
becomes operational and to identify any weak aspects of it. Although not 
all managerial problems can be identified during the build-up period, re- 
search at this stage can identify deficiencies and, at the same time, provide 
useful information to guide both the redesign of system components and the 
manner of their deployment. In an effort to maks those propositions explicit, 
v,e present a hypothetical example in the next section. 

4.2 What are the decisions, what information is needed? 

Let us assume that a program is begun to lengthen the months of infant 
breast-feeding among mothers in semi-urban areas. The behavioral model de- 
fined by planners assumes (and ideally the assumption has been verified) 
that the decision to curtail breast feeding is in many cases the result of 
a belief that bottle and breast-feeding are identical in their health conse- 
quences, and an attitude among women that their peers would look down on 
them if they continued breast feeding beyond a certain point. (We have no 

-29- 



data ourselves that suggols that this is an adequate model.) Planners 
might then desm'be an operational model which depends on a radio advertising 
campaign to change beliefs about the relative health benefits of breast and 
bottle feeding, and on local health aides to organize discussion groups 
of mothers-to-be to reinforce the radio message, and reverse both the 
perception and the reality of peer pressure to give up breast feeding. In 
Figure 1 we present part of an operational model that might have been 
specified by planners of such a program. 

Obviously many important activities are excltded, and *he activities 
described in each box could be broken down into linked sub-activities. How- 
ever, for the purposes of an example, this is sufficient. Each of the- arrows 
represents a hypothesized link between two activities the project wishes to 
realize. If one delivers radio spots and payments to selected radio sta- 
tions, the spots will be broadcast at the specified times. Each cf these 
links is subject to verification. Is it really true that if women listen 
to the spots regularly they will understand the content? Will group 
sessions with the health aides reduce women's perception of bottle feeding 
as prestigious? 

The typical evaluation, we have said, asks oi.ly "did it work?' 1 In 
this situation: did all tl ese activities as a whole, affect breast feeding 
behavior? The problem is that a negative answer io that question gives no 
guidance to project administrators. They do not know what to change. In 
contrast, attempts to verify each of the links in the system independently 
of the existence or contr.bution of each of the others, may enable admini- 
strators to pinpoint needed modifications. , 

While we say that the responsibility of the rasearcher is' to verify 
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Figure I 

PARTIAL MODEL FOR A BREASTFEEDING COMPAIGN: 
(adapted from materials used at the Latin 
America Conference on Nutrition Education) 
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70% of the Women 
Listen to the 
Spots Two Times 
Every Three Days 
For Two Months 



90% of the 
Women Who Listen 
To the Spots 
Understand the 
Sense/Content 
of the Messages 



Health Aides Suf- 
ficiently Trained to 
organize and to Con- 
duct Effective Meetings 



60% of the Women 
(Specific Audience) 
Participate in Meetings 



70% of the 
Women. Who Under- 
stand the Messages 
Change their Atti- 
tude Concerning the 
Relative Value of 
Breastfeeding vs. 
Bottle Feeding 



The Health Aides Put 
in Practice Their Ac 
quired Skills Giving 
the Women the Oppor- 
tunity to Discuss and 
Decide 
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feeding In- 
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Months 
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Change their 
Atti tude 
Concerning the 
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each of the links in the planner's model, we must first recognize the 
context within which such activity takeo place. The job is complicated by 
the rapid mobilization of material and human resources as production arid 
distribution activities get underway . Ihe organization of the field activ- 
ities is time consuming to the extent that it reduces enthusiasm for verification 
activities. As the scheduled inauguration day approaches, the pace of activit 
reaches a feverish pitch. New equipment is Installed, personnel activities 
at various levels must be coordinated, materials begin to flow in torrents 
from production centers to the field. The demands on program administrators 
are almost always overwhelming. 

Aside from the problem of Intensity there is a problem of time and re- 
sources. The links described in Figure 1 are only a small portion of the 
many thajt are essential to a project and thus worthy of verification. 
Literallif hundreds of discrete researchable questions can be identified, 
and the (staff and budget assigned to the evaluators can not possibly match 

those demands. 

i 

Whit is a researcher to do? First, avoid the temptation to withdraw 
during the tempestuous implementation oeriod, wait for the dust to settle, 
and only begin work when system operations have been stabilized and regular 
patterns of utilization have emerged. If the researcher's only task was to 
assess project impact, that might be reasonable. If it is to verify the 
usefulness of the operational model In detail, research must begin earli *\ 
Second, develop a 11st of the links which are crucial to project operations, 
and about which some doubt exists, if a link is deemed non-essential, or 
if all parties are confident of Its existence, verification will naturally 
have a lower priority. Priority must be assigned to actlvites which are at 
least potentially modifiable. Third, recognize that commitment of resources 
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to one research effort reduces resources available for others. Consider a 
range of research approaches, and adopt the lowest cost one which can pro- 
vide an acceptably credible answer. Research should be used to reduce un- 
certainty about a link, not eliminate uncertainty altogether. Finally, use 
strategies that permit looking at the link between two activities without 
requiring that all other activities be in place simultaneously. 

Two applications of the above criteria to the hypothetical breast- 
feeding campaign may provide useful illustration. Will the delivery of radio 
spots and payments to radio stations actually result in the broadcast being 
aired at scheduled times? One could verify that link by paying individuals 
to listen and keep a log of when the scheduled broadcasts begin. Or one 
can assume that broadcast stations will treat all paying advertisers alike, 
and then interview regular advertisers as to their experience. This second 
strategy entails some risk - regular advertisers may be treated differently 
than social advertisers - but it is much less expensive, and it may provide 
adequate indication of trouble ahead. By keeping the qosts of verification 
activities down, resources will be available for other efforts. Also, by 
obtaining the information before a campaign goes on the air nationally, 
modification may be possible before any damage is done. 

Verification that women will change their attitudes^ about the prestige y 
of bottle feeding as the result of group discussions with their peers may 
also be desireable. Furthermore, it would be worthwhile to verify such an 
activity independently of the related issues of recruitment of participants 
and of training of health aides to lead the discussions. While each of 
those activities are essential to the model, it would be helpful to be ' 
able to avoid confounding the three issues. To estimate the result of group 
discussion run according to plan, researchers might arrange a half doz£n 




prototype meetings with representative g. oups of the target population. 
Each would be led in the prescribed manner. Before-after testing, even with- 
out a control group, would provide some sense of the discussions 1 effectiveness. 
Ideally, long term effects ought to be estimated as well, but if they are 
expected to^pccur only in the presence of the reinforcing ad compaign, 
those efforts may be unobservable until actual implementation of the system. 
While this is not a low cost research strategy, it does permit some reduction 
of uncertainty about a given link, while allowing modification of early 
activities (like training of health aides) before too much has been invested 
in them. Of course, many verification activites will have to await fuller 4 
scale implementation - for example, the effectiveness cf recruitment activi- 
ties in bringing large numbers of women to discussion groups. The uncer- 
tainty reduced by a test under pilot conditions would be minimal. 
4.3 W hat are the Research Questions? 
Institutional issu es 

1. How can system-implementation research best be organized and 
applied? Hew can adequate human and material resources be mar- 
shalled to created such capacity within new communication projects? 
How can project administrators and staff be brought into the veri- 
fication process, both as sponsors, as definers of research prio- 
rities and as users 0. information. In sum, how can such studies 
be effectively 'inked to the behavior of administators and other 
program participants so that their decisions will be based on 
progressively more sophisticated understandings of the processes 
and resources they control . 

Obviously, there will be pressures which occasionally will send researchers 

off on tangents of their own, or close administrators' eyes to all but 

the logistics of program implementation. It is the definition of those 

pressure? which both establishes some limit on the practicability -f this 

model verification approach, and which suggests possible institutional 

procedures for living with them. This suggests the following research 
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approaches: 



a. There are a number of -rojects which have relied on research 
to help shape Implementation: SITE, Sesame St reet, Nicaragua Radio 
Math and The British Open University comTT^?Tately to mind. Such 
experiences might be contrasted with those of projects which either 
tried to Incorporate research and failed, or made no provision for 
Implementation research to begin with. The goal would be: to 
identify those forces which work against project implemented 
researcher collaboration; to gain a better sense of the utility 

of Information researchers can gather at this stage; and to pro- 
vide some proposals for how best to augment model-verification 
activity 1n the future. 

b. As a follow-up to these case studies, several direct efforts 
to realize researcher/lmplementer colloboration could help to de- 
fine further the value and limits of model verification research. 
By providing expertise and budget supplements to perhaps 3-6 pro- 
jects, Investigators would be better able to see just what 1s en- 
tailed 1n developing this capacity, and 1n what circumstances 1t 
has real value. 

Methods Issues 

In general, the Issue here is: what are the appropriate methods for 
doing model verification research? The range of design and measurement alter- 
natives is broad. But constraints of time and money 1n the context of a large 
nunber of links worth verifying produce a need for minimal cost research 
strategies wh1t> still achieve adequate credibility. This demands considerable 
metnodological judgment and sophistication. Abandonment of cookbook criteria 
for selecting research designs burdens the researcher with the task of judging 
the loss in inferential power associated with given design and measurement 
compromises. Some would argue that this need for methodological sophistication 
makes the adoption of the model verification approach impracticable in m?ny 
applications. They may be right, however it may be possible to lessen the 
need for such judgments or compromises 1f weU-tested alternative approaches 
to verifyin :ommonly observed links in typical operational models are avail- 
able, Message and materials pretesting is one such well-established area. 
Another is establishing listenership levels. There may be methodological 
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problems which are common to more than one verification issue. For example, 
obtaining representative samples of the target population is frequently neces- 
sary. Observation of group meetings or classroom activity is commonly re- 
quired. For these typical verification and methodological problems and others 
like them, there are a range of approaches of varying cost available. 

?. The research question is what quality of answer can be bought 
for what resource expenditure. The goal, while respecting differ- 
ences in applicability across different situations would be a 
guidebook organized around typical research questions and describing 
a range of research approaches with assorted estimates of their 
usual validity and cost. 

To illustrate th*s research issue we return to the example of pretesting 
materials. Rare is the communication plan that does not call for some 
increase to be made in the amount of information available to some sector 
of a society or to the society at large. The messages may be of various 
kinds, verbal as well as non-verbal, and they may be organized or "packaged" 
in a variety of ways. No producer, no matter how expert or experienced, 
can be sure that the message* he or she produces will have the effect in- 
tended on ar audience unless they have been tried out on members of that 
o'idience. Va.-ious techniques have been used to pretest messages before 
distributing them to a large audience. Peer ratings are a natural and effec- 
tive way to ju^ge whether or not messages fulfill the expectations of fel- 
low producers and are comprehensible to them. If the messages are not com- 
prehensible to this group, the odds are slight that they will be compre- 
hensible to a wider audience. The attractiveness and motivating Potential 
of messages can also be rated by outside experts, but here judgments must 
be considered only partially valid. The levels of understanding and the 
aesthetic sensibl ilites of a program producer or media critic are likely 
to be far more acute than the average audience member's, rendering the 
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former's assessment of message appeal or potential impact only partially 
useful as a guide to future program development. 

One way program formats have been pretested without incurring excessive 
expense is by actually presenting prototypes to representatives of the 
potential audience, either at production headquarters or in the field. 
The evaluation unit of the Children's Television Workshop, for example, 
developed a variety of studio-based techniques to pretest program formats 
for both Sesame Street ard the Electric *npany . Such measures included 
physiological as well as cognitive and attitudinal indicies o* message 
appeal and comprehension. The information gathered allowed television 
producers to reformulate program formats and modules more closely attuned 
to the tastes and viewing habits of their young audiences. 

The evaluators of SITE tn^d to carry the Sesame Street model one step 
farther by actually pretesting their programs in rural Indian villages. To 
do this, they transported videotape recorders to the field to reach potential 
audience members in as natural a setting as possible (Mody, 1976). Regret- 
ably, the logistical requirements of moving cumbersome and expensive studio 
equipment into the field undermined the naturalness of the test and, one 
suspects, the validity of the results as well. However, the strategy was 
worthwhile, for it did call attertion to the need for simpler pretest 
strategies that could simulate as much as possible the viewing conditions . 
that SITE broadcasts were likely to encounter -fn the villages. 

We would suggest two approaches to realizing this guidebook to alter- 
native research strategies: 

c. The f ; rst task is codification of existing experience. By 
a review or cases - both through the published literature, and 
interviews with individuals involved with given projects - it 
would Lv possible to describe the research questions that repeat 
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across projects. How to do pretesting, how to estimate listen- 
ership, etc.? As part of this exercise, descriptions of alter- 
native mechanisms used in each project could be developed. 

d. The codification of existing experience might be followed 
by a direct effort to explore, in a consistent fashion, the 
utility of alternative approaches within ongoing projects. By 
supplying expertise and budget supplements, so as to make pos- 
sible more research activity than would be ordinarily feasible 
oV requires, researchers would be able to compare directly three 
or four strategies for answering specific questions. It should 
be possible to detail the activities required for each, the costs 
they entail, the type of results they obtain, their true validity 
(measured against an ideal baseline procedure, perhaps) and the 
demonstrated utility of their results to program adminstrators. 
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5.0 SYSTEM MAINTENANCE 

5.1 W ho are the actors? 

The implementation of a communication system on a regional or 
national scale inevitably requires the active cooperation of people in 
many different localities. Such cooperation is necessary not only to launch 
new programs, but also to keep them running smoothly. The sheer physical 
distance separating the producers and the receivers of messages can create 
delays and misunderstandings. So, too, can the lack of direct contact 
between the various actors in the project (content specialists, mass media 
experts, extension agents, etc.) who have been linked to one another and 
to the client group through various iterations of the system model dis- 
cussed above. The assumption that there •-'11 be a unity of interests 
among all potential participants in a new communication system is a naive 
one and such thinking can severely undermine operations as they expand to 
cover larger geographic areas. 

Virtually all communication systems inaugurated in recent years have 
supported evaluation units o^ one kind or another. The ostensible purpose 
of such units has been to aid decision-makers through the provision of re- 
liable data on program p?rformance. Yet, the real value of such activity 
depends on the administrators' ability to anticipate potential problems and 
to commission research which examines the precise nature and causes of 
such problems. The aggregation and interpretation of feedback within most 
communication systems continues to be of a rather mechanistic and purposeless 
character, occasionally permitting administrators to identify problems, but 
only rarely providing the kinds of information necessary either . v o make in- 
ferences regarding the performance of individual system componencs or to 
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illuminate ways to overcome their weaknesses in the field. 

The users of research and evaluation for system maintenance are not 
substantially different from those at the orevious build-uj phase. How- 
ever, even if the actors are the same, their concerns are different. At 
this stage, the project has an operational model , one whose components, we 
hope, have been verified as they were installed. In any case, some system 
is working, and flexibility is likely to be limited. The goal for this 
phase is fine tuning and not, at least in the short run, major shifts in 
the role of major system components. 
5.2 What are the decisions? What information is needed? 

There are really three types of decisions being made once a project has 
reached the maintenance stage. First, there is the correction of foul-ups 
in project logistics: delays in the delivery of materials, arrangement of 
group sessions or maintenance of television or radio receivers. Second, 
there is the restructuring of particular components ( short of their elimi- 
nation) when they no longer are serving adequately: slowing the pace of 
broadcast instruction or changing the incentives for participation in the 
field. Finally, 1t may be necessary to augment or re-assign responsibilities 
within projects - for example, the development of new content materials or 
the incorporation of new audiences ( rural mothers as well as semi-urban ^ 
ones). Each category of decision demands a differed research f *ck^ 

The correction of logistical problems is the only one cf the three 
decision domains likely to be well served by feedback - that 1s, by the 
regular gathering and reporting of information from the field. The centri- 
fugal character of communication projects often does cause administrators 
to lose control of vital components as they expand in size. It *s imperative, 
'nerefore, that feedback strategies be installed to insure that all planned- 
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for events in the system model are actually taking place. Are the receivers 
working? Are the group sessions taking place? Are the materials arriving 
on time? Delivery mechanisms merit constant monitoring because they are 
likely to go wrong at any time. The monitoring usually can be done at 
reasonable cost and logistical failures may often be correctable wi' lout 
major increases in project cost. 

Constant monitoring of a svstem's logistical or delivery components is 
justified because such information can be used. This is net true for infor- 
mation designed to influence the second type of decision. 

Restructuring the role of a project component - slowing down the pace 
of mathematics instruction in broadcast lessons, for example - may also seem 
to justify constant monitoring. After all, constant knowledge about the 
level of mathematics learning would leave a project staff ready to deal with 
any shortfall as it occurs. However, for most developing country educational 
media systems, such regular monitoring would be both expensive, time con- 
suming and of only marginal utility. Obtaining learning data six times a 
year on each of five broadcast subjects for each of five grade levels (as 
would have been required in a feedback system piloted in El Salvador) would 
require a minimum of five professional evaluates and a support staff of 
twenty. Such a stai f could not have been employed, r or to do so would have 
left no time for the myriad other research tasks of equal importance. 

Since production lead time in educational media systems is, or ought 
to be, «ong, short term remediation capacity is quite limited. In other 
words, the potential for correcting observed weakness in the short run is 
small. Furthermore, knowing that learning is inadequate is not the same as 
knowing how to correct it. Guidance in locating a solution may require 
additional follow-up research. Finally, constant monitoring of learning 
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presupposes that learning success wi 1 V vary from year to year for reasons 



build-up phase one had verified the effectiveness of the mathematics course. 
It is possible that as successive cohorts enter an instructional system, 
their background characteristics will change, and their capacity to make 
use of already- prepared programs will be reduced. Yet, such cohort change 
is likely to be slow. It would scarcely justify an expensive monitoring 
system to verify the effectiveness of an instructional unit in year two 
if that same unit had already been verified in year one. 

In lieu of constant monitoring, component-restructuring decisions are 
likely to be best served by specially-mounted research projects designed 
to investigate particular system components. Such studies should be designed 
to estimate the effectiveness of the components, and to define just what, if 
anything, is wrong with them. They would be triggered on an irregular basis, 
either by suspicions of the project staff, by knowledge that changes in 
an audience make continuing effectiveness uncertain or, ideally, by a low 
level monitoring mechanism njaking use of informal feedback from field per- 
sonnel. There is a risk that these triggering mechanisms would be inade- 
quate and leave important problems undiscovered, That risk would be exacer- 
bated if the materials production staff had little or no contact with 
field operations. In such situations, some regular investigation of 
essential project components might be necessary. Hoewever, the intervals 
between such investigations could be long. 

The third type of decision - incorporation of new responsibilities - 
is not essentially different from the activities undertaken in the system 
build-up phase. As such, it is not a new issue and will not be considered 
further in this section. 
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5.3 What are the Research Questions? 
5.3.1 Institutional Issues 

1. Which system components justify constant monitoring;, which ones 
will only benefit from irregular attention? 

2. How can a project be made maximally open to new information? 
it, part, this question asks what sort of information is likely 
to be most useful and credible with project staff and admini- 
strators; in part, it asks how can the leadership roles be struc- 
tured so that they have flexibility, time and incentive to respond 
to information from the field. 

Obviously, the answer to each of the above questions depends on the answer 
to the other, and each, in turn, will reflect the circumstances of a par- 
ticular project. Implicitly, they call for a test of the hypothesis that 
regular monitoring of sys*«an effectiveness is not useful, that once effec- 
tiveness is established in the build-up phase, intensive study of parti- 
cular components on an irregular basis is likely to be more valuable. 
Methods for examining these questions will overlaD with those of previous 
sections. 

a. A survey of experience, with intensive study of those cases 
which have had operating research units, 1s again the first order of 
business. Such a survey would attempt to locate what kinds of 
research results were and were not used and what factors in the 
organization of a project aided or blocked the easy utilization 

of results. 

b. An attempt to reform existing research units - to serve both 
monitoring and special investigation needs - would be the next 
step., This effort could be associated with ongoing projects In 
exchange for providing training of in-house researchers and technical 
expertise, Investigators would request permission to observe, de- 
scribe and define answers to this section's c-uestlons. Projects 

to receive such aid and observation could be sampled from a ranqe 
of applications. 

5.3.2 Methods Issues 

The. methods questions at this stage are less novel than the Institu- 
tional issues. Essentially, they are identical in type to those proposed 
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for the system build-up phase. They involve alternative strategies for 
monitoring the implementation of system components and for focusing attention 
on special problems as they arise. The list of researchable questions in 
both areas would come from the activities previously described in this section. 




6.0 SYSTE M REVIEW 

6.1 Wha are the actors ? 

In the introduction to this paper wc argued that summative evaluation 
questions (e.g. "Did the system accomplish its objectives?" Did it 
succeed?") do not command much attention among administrators of on-going 
projects. Were the answers to such questions m^de available in a timely 

y 

fashion, and were they t<* contain information which bore directly on the 
performance of one or more system components, adminstrators might consider 
taking corrective action. Generally speaking, however, administrators do 
not have the flexibility (or the willingness) to respond to summative judg- 
ments concerning their own system's impact. Who, then, |s concerned with 
overall questions of system impact? 

We can identify three groups of decision-makers who are or should be 
concerned with system impact Information. First, there are funding agency 
officials and policy planners who are explicitly concerned with replication 
and technology transfer. What does the experience with mathematics instruction 
by radio in Nic^qua suggest for solving comparable educational problems 
in Thailand or in the Philipines? Secondly, tnere are policy makers who are 
considering the expansion of pilot projects. What does the pilot experience H 
with mathematics education by radio suggest about the promise of a nation- 
wide radio mathematics program? Finally, there are newly appointed policy 
makers, individuals without commitment to specific projects, who may wish 
to set a fresh agenda by radically altering or eliminating the programs of 
p/evious administrations. 

6.2 What are the decisions? 

For all three groups, the essential decision involved is whether or 
not tc sponsor a project in a new environment. The choices wi 11 rarely, if 
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ever, be to replicate an existing model exactly. Inevitably, new or expanded 
systems will operate under circumstances which are quite different from 
existing ones. The real question 1s what lessons does an existing system 
hold that are transferable to new environments. Obviously, the magnitude of 
environmental differences between existing and proposed projects will have a 
great deal to do with the transferability of experience. Furthermore, 
planners of systems in new settings will have more uncertainty than those 
who are working on extensions of existing ones. However, in both cases 
important ^nfferences will be required. One might argue, for example, that 
differences 1n setting and audience between the model system and some pro- 
posed new system are so minor as to present no obstacle to transferability. 
Or, mc.e commonly, one might argue that specified changes in the system's 
operational model will be sufficient to eliminate any problems caused by 
such setting and audience differences. In either case, an adequate under- 
standing of just how the effectiveness of an operational model is 'ikely to 
vary with changes 1n sotting and audience 1s required, it is not enough 
to know that a particular approach succeeded, even if it is accomplished by 
a description of the environment 1n which it worked. Somehow, evaluators 
who want to be helpful^ to future projects must be able to convey in some % 
detail what the nature of the interaction between treatment and environment 
was. 

This perspective. is reinforced when we recall that operating systems 
are a product of both 'objective, 1 technical decisions and negotiation among 
Interested parties. Planners of new systenls will not want to transfer un- 
necessary baggage. Nor will they want to adopt an e^istirg ruodel in toto 
1f some of Its components are not required 1n a particular setting. Just 
as surely, political forces 1n the new environment will shape the operational 
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model that is eventually implemented. In both situations, detailed 
knowledge of each component's role in the existing system will be neces- 
sary. 

6.3 What information is required? 

In seeking policy guidance from an existing system, the researcher 
must ask: how did it work, for whose benefit, at what cost, and under 
what conditions? The "how did it work 11 question reinforces the need to 
gather evidence parallel to what we referred to as model verification re- 
search in the system build-up section. Once a system oecomes fully opera- 
tional, evaluators may estimate the contribution of each of its components 
to overall success. Of course, in some sense this may be an impossible task; 
the effects of the various components are almost always confounded. Yet, 
one does not require precise estimates of ultimate contribution to be helpful, 
and it should at least be possible to determine if major components operatea 
more or less as originally planned. 

A second type of information may be used to address the 11 for whose 
benefit 1 ' question. There is likely to be a great deal of individual and 
group variation in reaction to educational media programs. While much of 
this may be idiosyncratic, some variation is almost surely related to iden- 
tifiable audience characteristics - skills in decoding new information, 
antecedent behaviors and the social and economic incentives supporting them, 
physical access to complementary information sources, etc. By pinpointing 
which segments of an audience respond to a project, research findings 
can provide planners with a tool for estimating the likely effectiveness 
of similar systems on different groups. An important aspect of this analysis 
would Involve examination of the background resources cognitive, attitudinal 
and material) participants brought to the project as w*n as the benefits 
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they received. 

The *Hird area of concern 1s th? of costs. Much has been written in 
recent years on estimating cost* ison, Klees and Wells, 1978; Eicher and 
Orivel, 1977) and for that rea*jn less attention will be required here. 
However, 1n parallel with previous comments, we note that straight-forward, 
"how much did the project cost" inquiries (even with appropriate social 
discount estimates built in) are limited in the same w>y that H P1d the project 
succeed?" kinds of questions are limited. Overall cost f inures are of little 
use in the planning of projects for new environments. Factor prices may 
be different U/.laries higher or lower, cost of capital higher), the mix of 
factors may change, and the time structure of utilization may be faster or 
slower. In the future, analysts should disaggregate costs as much as 
possible so that they can be scrutinized and, where necessary, recombined 
by planners to suit different circumstances. 

The fourth type of information useful for the planning of future 
projects is institutional in character. What kinds of leadership are 
required at different stages of a project; what mechanisms encourage coordi- 
nation between production and field utilization agencies; what management 
strategies optimize productivity; what institutional arrangements enable a 
project to achieve some permanence; how can responsiveness to participant 
concerns be maintained over the long run? Such questions are among the most 
6^ ficult to answer - particularly in the context of a single project. To 
begin with, there is the difficulty of adequately describing the institu- 
tional arrangements that characterize any single project. Official organi- 
zation charts, displaying the internal hierarchy and the external links 
of a project, ma> only hint at true organizational relationships *nd loyalties. 
Informants can report their perception of key institutional arrangements, 
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but here there is high risk of bias. 

Even were one able to describe them adequately, assessing the conse- 
quences of institutional arrangements' is a conceptually troublesome activity 
in the context of a single project. Even when a project is governed by a 
single set of institutional arrangements, there is no logical way, within 
the context of that project, of determining what would have happened had 
it been implemented differently. Troublesome rr not, this is essential 
information for planners of new projects. How to get it raises the next 
issue - the need to look beyond the borders of a single project. 

As long as we maintain the fiction that the administrators of on-going 
projects are the prime users of summative evaluations, single project studies 
will continue to dominate our field. Tha planners of new projects (and to 
a lesser xtent the extenders of old ones) do not really want detailed de- 
scriptions of projects. Rather, they wish to know what prior experience, 
accumulated across a number of projects, can suggest about how best to conduct 
a new project. In other words, they do not want evaluations of projects; 
they want evaluation of promising solutions to particular social problems. 
Admittedly, sometimes that information can come from single project evaluations 
More often, however, it will come from evaluations which explicitly compare 
projects. Institutional issues, as we have already suggested, ar- particu- 
larly well-suited to cross-project investigations. 
6 . 4 Wh at are tie research questions? 
6.4.1 Inst itutiona 1 Issue s 

1. What types of information derived from the experience of other 
projects can the planners of new projects actually use? Which 
decisions are essentially technical and likely to be influenced 
by credible data from those projects? Which decisions, although 
subject to political demands, will be made more reasonably if 
competing constituencies have a clear-eyed and dcta-based picture 
of the likely consequences of their actions? 
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This research question is similar to the second one posed in the 
policy definition section. It should result in suggestions for what kinds 
of information ought to be made available to future planners. Accordingly, 
an important element of this research would be a description of the rele- 
vance (as perceived by planners) of types of information derived from 
earlier projects. One suspects, for example, that information about 
capital costs is likely to be accepted as more locally relevant thso infor- 
mation about the tendency for production staffs to beccne isoiatec from the 
field. Evidence that indeed there is variation in perceive relevance 
would have two implications for future research: (1) a oe-emphasis on 
issues for which little credit is likely to be givon to externally-gathered 
information, and, concurrently, (2) an affirmation of the need for rigorous 
cross-case comparisons and replications which do provide such information 
perceived as relevant. 

The second institutional research question assumes that the cross- 
project evaluation perspective of this paper is adopted. The problem 
then is putting it into practice. 

2. What^ institutional mechanisms are available for funding cross- 
project evaluations? Funding for evaluations is most often tied to 
particular projects - often because funders want an accounting. Are 
there likely funders for genuine cross-project evaluations? What 
are the forces that keep funders' eyes on the project at hand? Will 
those forces mean that all cross-project evaluation is post hoc - 
aggregating independent evaluations to make inferences? 

3. What institutional mechanisms are there for planning cross- 
project evaluation? How can an agenda of evaluation questions be 
generated, so that whether cross-project evaluation is done directly, 
or by aggregation of independent evaluations, there is some assurance 
that evidence will accumulate and address significant planning issues? 

It would be helpful to have some review of previous evaluations, with 

an eye to the function they served to their funders. Previous attempu at 

cross-project evaluation would be particularly informative. Interviews 
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with major 'natural 1 funders of cross-project research (foreiqn assistance 
agencies, international organizations, and foundations) could qive a richer 
oicture of tne constraints on such research. However, specific answers to 
questions about optimal future institutional mechanisms are likely to be 
the product of speculation and not direct research. 
6.4.2 Methodological Issues 

We conclude by raising three sets of essentially methodological questions, 
the answers to which should help media researchers and planners work more 
compatibly in the future. 

4. Is it feasible to isolate the effects of particular project 
components or, more specifically, their interactions with aspects of 
of the setting in which they occur? At what level can such isolation 
take place; is it restricted to a few very general , cateqories of ^ 
characteristics (e.g. groups versus no grcups, coordinating agency 
versus 'ingle implementing agency) or is there some jjossiblity 

of working with better differentiated categorie$,-faudience groups 
led by local volunteers to reinforce independently broadcast radio 
messages versus no reinforcing leaders in the same circumstance)? 

• 0 

5. What are the limits of cross-project generalization? In other 
words, to what extent can the goal of cross-project research -improving 
the next project down the line - be realized better in the context 

of single project versus multiple studies project. While institutional 
arrangements may be unique to a given project, and thus not pro- 
fitably investigated within those confines, other research questions - 
such as the interaction of audience character istics and component 
effects - may be usefully examined within a project. Obviously, 
there are advantages to replicating results across different projects. 
To what extent they outweigh the utility of results from intensively 
studied single projects is an empirical question. 

6. What are the appropriate methods for doing cross-project evalua- 
tion on such planning issues as: a) participation of local groups 
in the definition of project goals, b) coordination of broadcast and 
non-broadcast activities, c) design of content for various media and 
d) orientations and training of field workers? On the one hand, 
methods can include quantitatively coded characterizations or descriptors 
gathered from a large sample of projects which are used to predict out- 
put (see Morss et al , 1976, for an example of this). On the other 

hand, a series of independent qualitative evaluations may be used 
by experienced pase study analysts to make informed judgments. In 
the middle are a series of independently sponsored evaluations which 
share a list of research questions and methodological approachers, but 
which continue to examine individual projects with some degree of autonomy. 
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Middle courses always sound best, and we cannot deny some preference 
for them. However, they are not without problems: funding may be difficult 
to secure for evaluations of this kind, agreement on specific research and 
methods questions may be difficult to reach, and equivalent implementation 
of research designs may be difficult to realize if independent institutions 
are made responsible for each sub-evaluation. Depending on the issue, 
methods for doing cross-project research may well vary. 

For the kinds of questions listed in issue #6 above, available research 
should be reviewed. Problems of mr.king inference from available data should 
be defined and should lead to a description of the additional data that would 
be required to make confident inference. With such judgments in hand, a 
group of researchers could develop a set of additional strategies for an- 
swering each of the specific questions. Some strategies ought to emphasize 
study of one or more projects intensively; others independent evaluations 
across a range of projects; and still others relatively superficial data 
collection from a large number of projects. It may be possible, a priori , 
to eliminate some approaches as not-promising, and make some judgment about 
the logical utility of others given past experience. 

The last step could involve direct comparison of the alternative stra- 
tegies. Obviously, this would be sharply limited in both the number of spe- 
cific research questions and tithenumber of alternative approaches to be 
investigated. Both questions and methods ought to be chosen to represent 
potential planning issues. The pruJjctivity of each approach, given its 
cost and yield in information perceived re^vant to new projects., could 
then be reviewed and debated at conferences of planners, project admini- 
strators and researchers. 
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EPILOGUE : Addi tjonal Issues Raise d at t he Workshop 

P*rHc.i|i<wts In the Workshop n ted the useful but limited perspective 
adopted by tho authors ol (hn work I mi impnr. Anthony Halm of the hritl*li 
Open University and Hernando Bernal of Colombia's Acclon Cultural Popular(AlPO) 
questioned the emphasis given educational media projects and, specifically, the 
evaluation activities recommended to help new projects reach some maintainable 
levels of acceptability and effectiveness. Such an approach, most participants 
seemed to agree, did not deal adequately with the complex institutional 
environments and circumstances out of which most projects emerge. Bates, 
especially, argued that there are important differences between evaluation 
activities undertaken in behalf of media projects a** + hose undertaken for 
larger, multi-faceted educational institutions. By way of example, he 
contrasted individual projects, which tend to have a limited life, no per- 
manent budget, and relatively fixed and narrow objectives with fully 
develop and multi-faceted media systems, such as the Open University, which 
differ substantially on such characteristics. 

While the functions of research and evaluation may resemble one another 
closely, whether one is project or institution-oriented, the weighting and 
timing of such functions may vary widely. So, too, may the needs and expec- 
tations of decision-makers within an institution during different stages 
of a project's life. Alan Hancock of the HEP underscored this point when 
he argured that since the underlying legitimacy of institutions are rarely 
threatened by individual project evaluations, the leaders of those insti- 
tutions are usually more receotive to evaluation data that implies a major 
reformulation of project elements than are the administrators of those 
projects who must cope with a myraid of day to day pressures. 
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Whereas a number of participants suggested other ways in which the 
functions of evaluation and research differ with regard to institutional 
versus project requirements, Thomas Cook of Northwestern University and 
Emile McAnany of the University of Texas pointed out that the project- 
focused evaluation activities proposed in the working paper should be viewed 
as but one approach to the problem of estimating the value of various 
educational interventions or reforms. Cook then identified various types 
of evaluation. The most familiar type, and the one that received most 
attention 1n the working paper he noted, was the single project evaluation. 
At a slightly more abstract level is the evaluation of programs. It custo- 
marily tries to determine whether or not a broadly defined way of solving 
a particular educational problem (teaching primary mathematics via radio, 
for example), is effective across a number of environments. This, in fact, 
was the approach advocated in the last section of the working paper, the one 
dealing with Impact evaluation. At an even greater level, of abstraction, 
there Is the evaluation of entire sectoral programs: primary education, 
agricultural extension, rural health care, etc. According to Cook, it 
recognizes the inevitable consequences of heterogeneous implementation and 
utilization patterns and, at the same time, seeks to determine whether or 
not coarsely defined program initiatives can reach Intended audiences and be 
effective (or even survive) under widely varied local conditions. Finally, 
there is policy evaluation. It is concerned with the utility (the predictive 
and explanatory power) of the theories which underlay, either explicitly 
or implicitly, particular programs. At the outset of the working paper, 
such policy evaluation and review was advocated as the first essential 
step to effective project planning. If adequately conducted, such research 
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can provide a vital feedback loop and, at the same time, encourage decision- 
makers to reflect critically on the assumptions as well as the results 
of previous projects. 

last major critique of the Workinq Paper put forward by the parti- 
cipants was the authors' incomplete appreciation of the nature of the decision- 
making environment within agencies responsible for developing and admini- 
stering educational media projects. Clifford Block of the U.S. Agency for 
International Development expressed the view that most of the decision- 
makers' work is concentrated not on the formulation of policy, but rather on 
the "practical context" of making 'the best use of a very limited supply of 
resources." If, as most participants seemed to agree, the bureaucratic 
environment out of which educational media decisions flow is governed more 
by the competition for resources than by the systematic enactment of policy, 
the researchers' task is made even more complex and challenging than the 
paper indicated. Joao Oliveira of Brazil s Financiadora de tstudios e 
Projetos emphasized this point, arguing that the key fo understanding all 
educational media project? lay in their administrative environments. 
Survival is the guiding interest of most bureaucracies, he claimed, and all 
evaluation activities must be guided by that perspective. 

Given the complexity of the decision-making environment, can 
researchers ever be sure that their efforts are appreciated or even taken 
seriously? The participants seemed to agree that research and evaluation 
rarely affect projects directly. A far more familiar phenomenon is the 
evaluator's complaint that so little attention is paid to his or her 
efforts. While such a lament seems justified, at least in the short run, 
it ignores the many ways research and evaluation can and do affect projects 

/ 
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Indirectly. Over time, and across projects, results accumulate. Although 
planners may not make any direct reference to previous projects or their 
evaluations, their decisions about new projects do seem to be influenced by 
what has gone before. Such influence is transferred subtlely, through 
internal memoranda and, even more importantly, through the interpersonal 
communications of people who have been active in a given field for some time. 
Information transfered in these ways may become seriously distorted and/or 
overly generalized, but the effects of its influence are apparent nonetheless. 
For this reason, a better understanding of the political and administrative 
incentives which drive decisio -makers could aid evaluators immensely, both 
in the design of their studies and in the selection of more appropriate 
strategies for conveying their findings. 
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Communication Centre 
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Blackrock 
County Dublin 
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Geonje PAPAG IANNIS 
College of Education 
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Florida State University 
Tallahassee, Florida 32306 
(Tel: 904-644-4583) 
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APPENDIX B: WORKSHOP AGENDA 



Sunday, Oct, 26 



Participants arrive 



Monday, Oct, 2 7 



9:00 AM 
(Keen Bldg. - 7th fir.) 



Introductions and Orientation 
Alan Hancock and John Mayo 



10:30 AM 



11:00 AM 



Review of Discussion Paper, "Research Priorities 
for the Planning of Educational Media" 
Bob Hornik and John Mayo 



12:30 PU 



Lunch 



2:00 PM 



Evaluation as a Policy Instrument 

Animateur: Emile McAnany 

Resource persons: Cliff Block 

Tom Cook 
Joao Oliveira 



3:30 PM 



Retf/teahmeittd 
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4:00 PM 
6:30 n 
tt 10 PM 



Discussion 

Reception 
VinneA 
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Tuesday, Oct. 2<i 



9:00 AM 



Issues in System Planning fcvaluation 



Animateur: 
Resource persons: 



Hernando Bernal 
Alan Hancock 
Heather Hudson 
Bob Morgan 



10:30 AM 



11:00 AM 



Discussion 



12:30 PM 



Lunch 



2:00 PM 

Starry Conf. Rm. 
(220 Bus. Bldg.J 



Formative Evaluation of Educational Media 

Animateur: Roy Col le 

Resource persons: Tony Bates 

Bella Mody 
John Tiff en 



3:30 PM 



4:00 PM 



Discussion 



7:00 PM 
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9:00 AM Issues in System Impact Evaluation 

% Animateur: Doug Goldschnrdt 

Resource persons: Dennis Foote 

Dick Martin 
Francois Orivel 



10:30 AM Coiitt 

11:00 AM Discussion 
12:30 PM Lunch 

2:00 PM Elective Discussion Groups 

A. Policy Definition and System Planning 
Research Priorities 

Rapporteur: Joao Oliveira 



B. System Build-up and Maintenance: 
Research Priorities 

Rapporteur: Barbara Searle 



System ImDact and Costs: 
Research Priorities 

Rapporteur: Steve Klees 



B:00 PM VinneA 
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Thursday, Oct. 30 



9:00 AM Elective Group Reports and Recommendations 

UilO AM Co^ee 

11:00 AM Concluding Session: Recommendations tr HEP 

for Future Planning, Research, and Training 
Activities 

Friday, Oct. 31 Participants depart 
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